Semantic input sampling for explanation (sise) of convolutional neural networks

ABSTRACT

Embodiments of the present disclosure relate to generating explanation maps for explaining convolutional neural networks through attribution-based input sampling and block-wise feature aggregation. An example of a disclosed method for generating an explanation map for a convolutional neural network (CNN) includes obtaining an input image resulting in an output determination of the CNN, selecting a plurality of feature maps extracted from a plurality of pooling layers of the CNN, generating a plurality of attribution masks based on the plurality of feature maps, applying the generated attribution masks to the input image to obtain a plurality of visualization maps, and generating an explanation map of the output determination of the CNN based on the plurality of visualization maps.

CROSS-REFERENCE TO RELATED APPLICATIONS

Pursuant to 35 U.S.C. § 119(a), this application claims the benefit ofearlier filing date and right of priority to Korean Patent ApplicationNo. 10-2020-0103902, filed on Aug. 19, 2020, the contents of which areall hereby incorporated by reference herein in its entirety.

FIELD

The present disclosure relates to generating explanation maps forexplaining convolutional neural networks through attribution-based inputsampling and block-wise feature aggregation, also referred to herein asSemantic Input Sampling for Explanation (SISE).

BACKGROUND

Convolutional Neural Networks (CNN) have become a highly useful tool forartificial intelligence tasks such as image classification andprocessing. Deep Neural models based on CNNs have rendered inspiringbreakthroughs in a wide variety of computer vision tasks. However, thesemodels are often limited to outputting a classification or processingresult with little to no explanation as to how the result was obtained.Thus, the lack of interpretability presents a great obstacle in theunderstanding of the decisions made by these models. This furtherdiminishes the trust consumers have for CNNs and Artificial Intelligencein general, and may hinder the interactions between users and systemsestablished based on such models.

As an emerging field in Machine Learning, Explainable AI (XAI) attemptsto interpret these cumbersome models. The offered interpretation abilityhas put XAI in the center of attention in various fields, especiallywhere any single false prediction can cause severe consequences (e.g.,healthcare) or where regulations force decision-making systems toprovide users or systems relying on such models with understandableexplanations (e.g., criminal justice). This field aims to visualize thebehavior of models trained for image recognition tasks.

To achieve visual explanations for CNNs, methods based on classactivation mapping and randomized input sampling have gained greatpopularity. The outcome of these methods is an “explanation map”, whichis an image of a heatmap having the same size as the input image. FIG. 1depicts examples of explanation maps 101, 102 generated based on aninput image 100 that is input to a classification CNN. Based on adetermination by the CNN that the input image includes an image of a“Horse” 103, with a confidence level of 0.9956, the existing methodsgenerate explanation maps 101 indicating the levels of contribution ofvarious portions of the input image to the final determination of“Horse.” Based on a determination by the CNN that the input imageincludes an image of a “Person” 104, with a confidence level of 0.0021,the existing methods generate explanation maps 102 indicating the levelsof contribution of various portions of the input image to the finaldetermination of “Person.” The “hot” regions of the heatmap, denoted bythe red/yellow colored regions represent the portions of the input imagewhich have the greatest impact on the model's decision, while the “cold”regions of the heatmap, denoted by blue/green colors represent theportions of the input image which have little to no impact on themodel's decision making.

Prior works on visual explainable AI, such as those shown in FIG. 1, canbe broadly categorized into ‘approximation-based’,‘backpropagation-based’, ‘perturbation-based’, and ‘CAM-based’methodologies. In backpropagation-based methods, only the localattributions are represented, making them unable to measure globalsensitivity. CAM-based methods operate in three steps: 1) feeding themodel with the input image, 2) scoring the feature maps in the lastconvolutional layer, and 3) combining the feature maps using thecomputed scores as weights. Despite the strength of CAM-based methods incapturing the features extracted in CNNs, the lack of localizationinformation in the coarse high-level feature maps limits such methods'performance by producing blurry explanations. Also, upsamplinglow-dimension feature maps to the size of input images distorts thelocation of captured features in some cases.

These drawbacks are addressed by image perturbation techniques used inrecent works such as RISE (Petsiuk, V.; Das, A.; and Saenko, K. 2018.RISE: Randomized Input Sampling for Explanation of Black-box Models),and Score-CAM (Wang, H.; Wang, Z.; Du, M.; Yang, F.; Zhang, Z.; Ding,S.; Mardziel, P.; and Hu, X. 2020. Score-CAM: Score-Weighted VisualExplanations for Convolutional Neural Networks). These visualexplanation methods probe the model's behavior using perturbed copies ofthe input. Most of the perturbation-based methods' noticeable propertyis that they treat the model like a “black-box” instead of a“white-box.”

As an example of a random sampling method, FIG. 2 shows an input image201 to be input to a CNN model 204 for classification. FIG. 2 showsexamples of random perturbation masks 202 which are implemented in theRISE method existing in the prior art. The black mask portions of theperturbation masks are randomly generated, and the input image isperturbed, or masked, with the various perturbation masks such that onlythe regions of the input image corresponding to the white areas areremaining in the perturbed images 203. The perturbed images 203 are theninput to the CNN 204, and the resulting classification and confidencescore is used to determine which regions of the input image contributethe most to the CNNs decision based on the corresponding perturbationmask—resulting in an explanation heatmap 205 as discussed with respectto FIG. 1.

However, these existing procedures involve feedforwarding severalperturbed images, which makes them very inefficient and slow. They alsosuffer from instability as their output depends on random sampling orrandom initialization for optimizing a perturbation mask. Also, suchalgorithms require an excessive runtime to provide their users withgeneralized results. Further, explanation maps produced by CAM-basedmethods suffer from a lack of spatial resolution as they are formed bycombining the feature maps in the last convolutional layer of CNNs,which lack spatial information regarding the captured attributions.Thus, the attribution methods based on the existing methods provideinefficient, low-resolution, and blurry explanation maps that limittheir explanation ability.

SUMMARY

To address the above issues, visualization and explanation of CNNdecision making based on various layers of the CNN is provided.Visualization maps from multiple layers of the model are collected basedon an attribution based input sampling techniques and the visualizationmaps are then aggregated to reach a fine-grain and complete explanationmap. A layer selection strategy of the CNN is provided that applies tothe whole family of CNN-based models to visualize the last layers ofeach convolutional block of the model.

Embodiments of the present disclosure provide visual explanationalgorithm specialized to the family of CNN-based models. The presentdisclosure includes a discussion of attribution-based input sampling andblock-wise feature aggregation, also referred to herein as SemanticInput Sampling for Explanation (SISE), which generates explanations byaggregating visualization maps obtained from the output of convolutionalblocks through attribution-based input sampling. Embodiments of thepresent disclosure output high-resolution explanation maps which faroutperform those of the existing art, resulting in a greater level ofinsight into the decision making of a CNN provided to users.

Embodiments of the present disclosure include systems, methods, andcomputer-readable media for generating an explanation map for aconvolutional neural network (CNN) through attribution-based inputsampling and block-wise feature aggregation. An embodiment of a methodof the present disclosure for outputting an explanation map for anoutput determination of a convolutional neural network (CNN) based on aninput image includes extracting a plurality of sets feature maps from acorresponding plurality of pooling layers of the CNN, obtaining aplurality of attribution masks based on subsets of the plurality of setsof feature maps, applying the plurality of attribution masks to copiesof the input image to obtain a plurality of perturbed input images,obtaining a plurality of visualization maps based on confidence scoresby inputting the plurality of perturbed copies of the input image to theCNN, and outputting an explanation map of the output determination ofthe CNN based on the plurality of visualization maps.

In another embodiment, the method further includes identifying the mostdeterministic feature maps with regard to the input image of each of theplurality of sets of feature maps as a corresponding subset of featuremaps.

In another embodiment, identifying the subset of feature maps which aremost deterministic with regard to the input image comprises calculatingan average gradient of the model's confidence score with respect to theinput image for each feature map. In yet another embodiment, a featuremap is selected as most deterministic if its corresponding averagegradient is greater than zero.

In another embodiment, the method further includes upscaling the featuremaps of each subset to an original size of the input image to generatethe plurality of attribution masks. In another embodiment, generatingthe plurality of attribution masks further comprises performing lineartransformation on the attribution mask that normalizes the values in therange of [0, 1], and applying the generated attribution masks to thecopies of the input image comprises performing perturbation of thecopies of the input image based on each of the generated attributionmasks by point-wise multiplication. In yet another embodiment,outputting the explanation map comprises performing a fusion process tocombine feature information from the plurality of visualization maps.

In an embodiment of a method of the present disclosure, the fusionprocess to combine feature information of two visualization maps of theplurality of visualization maps comprises: normalizing a firstvisualization map of the plurality of visualization maps; performingunweighted addition of the normalized first visualization map and anormalized second visualization map to obtain a first result; performingOtsu-based binarization on the normalized second visualization map toeliminate features which are not present in the normalized firstvisualization map to obtain a second result; performing point-wisemultiplication on the first result and the second result to obtain athird result; and performing the fusion process using the third resultand a next visualization map of the plurality of visualization maps. Incertain embodiments, the explanation map has a same dimensionality asthe input image.

In another embodiment of the present disclosure, a machine-readablenon-transitory medium having stored thereon machine-executableinstructions for outputting an explanation map for an outputdetermination of a convolutional neural network (CNN) based on an inputimage is disclosed, wherein the instructions comprise extracting aplurality of sets feature maps from a corresponding plurality of poolinglayers of the CNN, obtaining a plurality of attribution masks based onsubsets of the plurality of sets of feature maps, applying the pluralityof attribution masks to copies of the input image to obtain a pluralityof perturbed input images, obtaining a plurality of visualization mapsbased on confidence scores by inputting the plurality of perturbedcopies of the input image to the CNN, and outputting an explanation mapof the output determination of the CNN based on the plurality ofvisualization maps. In various embodiments of the machine-readablenon-transitory medium, the stored machine-executable instructions mayalso include various other features similar to those of the embodimentsof the method discussed above.

In another embodiment of the present disclosure, a system is disclosedfor outputting an explanation map for an output determination of aconvolutional neural network (CNN) based on an input image, the systemcomprising a display, one or more processors, and a memory havinginstructions stored thereon, which when executed by the one or moreprocessors, cause the one or more processors to extract a plurality ofsets of feature maps from a corresponding plurality of pooling layers ofthe CNN, obtain a plurality of attribution masks based on subsets of theplurality of sets of feature maps, apply the plurality of attributionmasks to copies of the input image to obtain a plurality of perturbedinput images, obtain a plurality of visualization maps based onconfidence scores by inputting the plurality of perturbed copies of theinput image to the CNN, and output, via the display, an explanation mapof the output determination of the CNN based on the plurality ofvisualization maps. In various embodiments of the disclosed system, thestored instructions may also cause the one or more processors to performvarious other features similar to those of the embodiments of the methoddiscussed above.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee. The above and other aspects and features of thepresent disclosure will become more apparent upon consideration of thefollowing description of embodiments, taken in conjunction with theaccompanying drawing figures.

FIG. 1 are illustrations of various explanation maps for an input imagegenerated according to the existing art.

FIG. 2 is an illustration of a method for generating an explanation mapfor a CNN according to the existing art.

FIGS. 3, 4A, 4B are illustrations of various aspects of a convolutionalneural network.

FIGS. 5A, 5B, 5C are illustrations of input images and attribution masksrelated to embodiments of the present disclosure.

FIG. 6 is an illustration of a cascading fusion module according toembodiments of the present disclosure.

FIG. 7 is a flowchart illustrating a method according to embodiments ofthe present disclosure.

FIGS. 8, 9, 10, 11, 12 depict testing results related to embodiments ofthe present disclosure.

FIG. 13 is an illustration of a device related to implementation ofembodiments of the present disclosure.

DETAILED DESCRIPTION

In the following detailed description, reference is made to theaccompanying drawing figures which form a part hereof, and which show byway of illustration specific embodiments of the present invention. It isto be understood by those of ordinary skill in this technological fieldthat other embodiments may be utilized, and that structural, electrical,as well as procedural changes may be made without departing from thescope of the present invention. Wherever possible, the same referencenumbers will be used throughout the drawings to refer to the same orsimilar parts.

Embodiments of the present invention relate to a system, method, andcomputer-readable medium for generating an explanation map for aconvolutional neural network (CNN) through attribution-based inputsampling and block-wise feature aggregation. While benefits of theinvention are discussed with respect to CNN implementations with respectto image processing and classification, the disclosure is not limitedthereto, and it will be appreciated that the invention is applicable tovarious implementations, including other CNN implementations such asaudio and/or other data processing and classification.

By way of background, a convolutional neural network (CNN) algorithmconsists of multiple convolutional layers, pooling layers, activationlayers, and fully-connected layers. FIG. 3 shows a generalized exampleof a CNN 300 comprising an input layer 301, convolutional layers 302,pooling layers 303, a fully connected layer 304, and an output layer 305which includes a single vector of probability scores used in artificialintelligence decisions, such as object or facial recognition.

In a convolution layer 302, the original input 301, such as input image100 of FIG. 1, is convolved with a set of filters (not depicted),otherwise referred to as kernels, designed to highlight specificcharacteristics of the input 301 to produce an output referred to as afeature map. While CNNs may be used in various applications such asimage processing or audio processing, this disclosure will use examplesrelated to image processing and recognition. However, it will beunderstood by those of ordinary skill that the disclosure may be appliedto CNN implementations for use in various input types and purposes.

The convolution layer 302 may include an activation layer (not depicted)to enhance the nonlinear property of the network by introducing anactivation function—such as a rectified linear unit (ReLU) activationfunction, wherein values that are less than or equal to zero become zeroand all positive values remain the same.

The pooling layer 303 produces a scaled down version of the outputfeature map from the convolution layer 302. This is achieved byconsidering small groupings of pixel regions and applying a desiredoperational filter across the pixel grouping to produce a singlerepresentation. Some examples of pooling layer implementations, notdepicted, include average pooling (computing the average of the elementspresent in the region of feature map covered by the filter), max pooling(selecting the maximum element from the region of the feature mapcovered by the filter), global pooling (reduces each channel in thefeature map to a single value, including global average pooling andglobal max pooling), and the like, although these examples are notlimiting.

Although not depicted in FIG. 3, most CNN implementations have multipleconvolutional blocks, where each block includes a pooling layer at theend. Typically, the number of convolutional block and pooling layercombinations in any given CNN implementation is five, however thedisclosure is not limited to this. The output scaled feature maps of themultiple convolution layers 302 and pooling layers 303 may be output tothe input of fully connected layers 304 comprising a deep neuralnetwork, having an input layer, hidden layers, and an output layer (notdepicted) for generating an output 305 for classifying the input image.The fully connected layer 304 provides a high-level reasoning byconnecting each node in the layer to all activation nodes in theprevious layer, and in many cases provides a traditional multi-layerperceptron neural network (MLP).

Based on the above background, details of the present disclosure followbelow, involving CNN-specific operations that improve the fidelity andplausibility (in the view of reasoning) of explanation maps whichprovide understandable visualization of the decision making of a CNN.Such operations are provided with adaptive computational overhead forpractical usage. The present disclosure does not rely on the randomizedinput sampling required in existing techniques such as RISE, aspreviously discussed. Instead, the present disclosure includes samplingthe actual feature maps derived from multiple layers of the CNN, wheresuch feature maps are “attribution-based”, indicating that it providesthe perspective of the model in various semantic levels.

In summary, the present disclosure includes four phases: in the firstphase, Feature Map Extraction, multiple layers of the model are selectedand a set of corresponding output feature maps are extracted. In thesecond phase, Feature Map Selection, for each set of feature maps asubset containing the most important feature maps is sampled. In thethird phase, Attribution Mask Scoring, the selected feature maps arethen processed to create sets of perturbation masks, referred to asattribution masks. The first three phases are applied to multiple layersof the CNN to output a 2-dimensional saliency map which is referred toas a “visualization map” for each layer. Such obtained visualizationmaps are aggregated in the fourth and last phase, Feature Aggregation,to reach the final explanation map.

Phase 1: Feature Map Extraction

In order to visualize multiple layers of the CNN to merge spatialinformation and semantic information discovered and utilized by theCNN-based model in its decision making, the most crucial layers forexplaining the model's decisions must be identified and extracted forvisualization. However, the sheer number of layers of a CNN have madethis task difficult in the implementations of the existing art.

Regardless of architecture, all types of CNNs consist of convolutionalblocks connected via pooling layers that aid the network to justify theexistence of semantic instances. Each convolutional block is formed bycascading multiple layers, which may vary from a simple convolutionalfilter to more complex structures (e.g., bottleneck or inverted residuallayers). However, the dimensions of their input and output signal arethe same. Each pooling layer decreases the computational complexity ofthe feature maps output by the convolutional layers. Contextually, thepooling layers perform an abstraction of higher-level features presentin the feature maps output by the convolutional layers by interpretingthem as shapes, objects, and textures. Each pooling layer typicallyreduces the dimensionality of the feature maps before outputting them tothe next convolutional block.

In detail, in a convolutional block, assuming the number of layers to beL, each ith layer can be represented with the function ƒ_(i)(.), wherei={1, . . . , L}. Denoting the input to each ith layer as y_(i), thewhole block can be mathematically described as F(y_(i))=ƒ_(L)(y_(L)).For plain CNNs (e.g., VGG, GoogleNet), the output of each convolutionalblock can be represented with the equation below:

F(

₁)=ƒ_(L)(ƒ_(L-1)( . . . (ƒ₁(

₁))))  Equation 1:

There are typically two types of convolutional networks. Plain CNNs,otherwise referred to as non-residual networks, refer to a typicallyshallow network where the output of each convolutional block isconnected to a pooling layer, and the output of the pooling layer isinput to the next convolutional block. These connections form a cascadedarchitecture. An example of a non-residual network is shown in FIG. 4A.In FIG. 4A, 401 and 402 are both convolutional blocks.

On the other hand, skip-connection CNNs, otherwise referred to asresidual networks, refer to a network where the connection between onelayer and the next layer are skipped, and/or connections exist betweendifferent convolutional layers. An example of an unraveled view of aresidual network is shown in FIG. 4B. As shown in FIG. 4B, the output ofa convolutional block 403, which is input to a pooling layer that is notdepicted, skips a connection and is instead connected at 405 to theoutput a different convolutional block 404. In such residual networksthat utilize skip connection layers to propagate the signals through aconvolutional block, these types of networks can be viewed by theunraveled perspective, and the connection between the input and outputmay be formulated as follows:

_(i+1)=ƒ_(i)(

_(i))+

_(i)  Equation 2:

and hence,

F(

₁)=

₁+ƒ₁(

₁)+ . . . +ƒ_(L)(

₁+ . . . +ƒ_(L-1)(

_(L-1)))  Equation 3:

The unraveled architecture as in FIG. 4B is comprehensive enough to begeneralized even to shallower CNN-based models that lack skip-connectionlayers. For plain networks, the layer functions ƒ_(i) can be decomposedto an identity function I and a residual function g_(i) as follows:

ƒ_(i)(

_(i))=I(

_(i) +g _(i)(

_(i))  Equation 4:

Such a decomposition yields a similar equation form as equation 2, andconsequently, equation 3.

_(i+1) =g _(i)(

_(i))+

_(i)  Equation 5:

It can be inferred from the unraveled view, as shown in FIG. 4B, thatwhile feeding the model with an input, signals might not pass throughall convolutional layers as they may skip some layers and be propagatedto the next ones directly. However, this is not the case for poolinglayers. Considering that a signal's dimensions are changed by a poolinglayer, Equation 4 cannot be applied to such layers.

Based on this information, a determination may be made that most of theinformation in each model can be collected by probing the output of thepooling layers. This has been further evidenced by experiments in whicherror rates of a CNN were observed as particular layers of the modelwere removed individually, one at a time. It was found that asignificant degradation in performance of the overall CNN resulted whena pooling layer was removed, where such degradation did not occur whenremoving a convolutional layer, showing that the performance and outputof the pooling layers played a significant role in the performance ofthe model's decision making and general accuracy.

Thus, it is determined that by visualizing the output of the poolinglayers, it is possible to track the way features are propagated throughthe convolutional blocks of the CNN. Therefore, in an embodiment of thepresent disclosure, the attribution masks for generating the explanationmaps are derived from the feature maps output by the pooling layer inthe last layer of each convolutional block for any given CNN.

Phase 2: Feature Map Selection

In the first phase, the model is fed with an input image to extract setsof feature maps from various layers of the model. In the second phase,Feature Map Selection, a subset of the most deterministic feature mapsamong each extracted set are selected, which are then processed toobtain corresponding sets of attribution masks, which are then used toperturb the input image in order to perform the “attribution-based inputsampling.”

In summary, identifying the most deterministic feature maps among theextracted sets of feature maps (output from the pooling layers, asdiscussed above), backward propagation is performed of the signal to aparticular layer to score the average gradient of the model's confidencescore to each of the feature maps. In this way, a value representing thecorrelation between each particular feature map and the original inputimage is obtained.

In an embodiment of the present disclosure, where the average gradientfor a particular feature map is zero or a negative value, it isdetermined that the effect of the feature map is insignificant and maybe disregarded, and thus the particular feature map is not selected forthe subset. On the other hand, where the average gradient for aparticular feature map is a positive value, it is determined that theeffect of the feature map is high, and the particular feature map isselected for the subset. However, the disclosure is not limited to this,and various other embodiments may include other thresholds for selectionand filtering of feature maps based on the corresponding averagegradient values.

Once the positive-gradient feature maps are obtained, the feature mapsare used to generate attribution masks by performing bilinearinterpolation, otherwise referred to as bilinear texture mapping. Asthese positive-gradient feature maps will all be dimensionally smallerthan the input image due to being output by the pooling layers, thepositive-gradient feature maps are also upsized to the same size as theinput image. This is followed by a linear transformation that normalizesthe values in the mask to generate the final attribution mask used toperturb the input image.

In detail, assume Ψ:I→

is a trained model that outputs a confidence score for a given inputimage, where I is the space of RGB images I={I/I:Λ→

³}, and Λ={1, . . . , H}×{1, . . . , W} is the set of locations (pixels)in the image. Given any model and image, the goal of an explanationalgorithm is to reach an explanation map S_(I,Ψ)(λ), that assigns an“importance value” to each location in the image (λϵΛ). Also, let l be alayer containing N feature maps represented as A_(k) ^((l))(k={1, . . ., N}) and the space of locations in these feature maps be denoted asΛ^((l)). These feature maps are collected by probing the featureextractor units of the model. The feature maps are formed in these unitsindependently from the classifier part of the model. Thus, using thewhole set of feature maps does not reflect the outlook of CNN'sclassifier.

To identify and reject the class-indiscriminative feature maps, thesignal is partially backpropagated to the layer to score the averagegradient of the model's confidence score to each of the feature maps.These average gradient scores may be represented as follows:

$\begin{matrix}{\alpha_{k}^{(l)} = {{\sum\limits_{\lambda^{(l)} \in A^{(l)}}\frac{\partial{\Psi(I)}}{\partial{A_{k}^{(l)}\left( \lambda^{(l)} \right)}}}:}} & {{Equation}\mspace{14mu} 6}\end{matrix}$

The feature maps with corresponding non-positive average gradient scores−α_(k) ^((l)), tend to contain features related to other classes ratherthan the class of interest. Such feature maps are referred to as“negative-gradient.” Instead, the set of attribution masks obtained fromthe “positive-gradient” feature maps, M_(d) ^((l)), is defined as:

M _(d) ^((l))={Ω(A _(k) ^((l)))|kϵ{1, . . . ,N},α _(k)^((l))>μ×β^((l))}  Equation 7:

Where β^((l)) denotes the maximum average gradient recorded.

$\begin{matrix}{\beta^{(l)} = {{\max\limits_{k \in {\{{1,\;\ldots\;,N}\}}}\;\left( \alpha_{k}^{(l)} \right)}:}} & {{Equation}\mspace{14mu} 8}\end{matrix}$

In equation 7, μϵ

_(≥0) is a threshold parameter that is 0 by default to discardnegative-gradient feature maps while retaining only thepositive-gradients.

Furthermore, Ω(.) represents a post-processing function that convertsfeature maps to attribution masks. This function contains a ‘bilinearinterpolation,’ upsampling the feature maps to the size of the inputimage, followed by a linear transformation that normalizes the values inthe mask in the range [0, 1].

FIGS. 5A and 5B depict examples of the generated attribution masksresulting from Phase 2's Feature Map Selection. The input image 501 isshown in FIG. 5A. Based on the above disclosure, the feature maps outputby the pooling layer in the last layer of each convolutional block ofthe model are selected in Phase 1. Then in Phase 2, the signal ispartially backpropagated to a particular layer of each feature map toscore the average gradient of the model's confidence score to thecorresponding feature map in order to obtain a subset of thepositive-gradient feature maps for each pooling layer output. The subsetis processed and upsampled, and attribution masks such as the examples502-506 of FIG. 5B are generated.

As shown in FIG. 5B, the attribution masks may have detectable featuresat various levels of abstraction depending on the pooling layer of theextracted corresponding feature map. Since the convolutional layers andpooling layers serve to further abstract the various edges, shapes,textures, and the like, with each convolutional block, the attributionmasks based on feature maps extracted from a fifth or fourth poolinglayer, such as 506, 505, respectively, may have very littledistinguishable features of the input image 501, in comparison toattribution masks based on earlier pooling layers such as 502, 503.

By contrast, FIG. 5C depicts examples of perturbation masks 507 utilizedfor methods of the existing art, such as RISE, previously discussed. Itcan be seen in comparison that the attribution masks 502-506 of thepresent disclosure, when applied to the input image 501, will result inperturbed input images to the CNN for explanation which will leave themain features, edges, shapes, textures, etc. of the input image 501unmasked, thereby resulting in more accurate and efficient explanationmap generation as compared to the randomly generated perturbation masks507.

Phase 3: Attribution Mask Scoring

Based on the attribution masks output from Phase 2 above, theattribution masks are then applied to copies of the input image using apointwise multiplication process, and the perturbed images are input tothe CNN to generate visualization maps based on their respectiveconfidence scores. Specifically, the first three phases are applied tomultiple layers of the CNN to output a 2-dimensional visualization mapfor each layer.

In detail, considering the same notations as the previous section, theconfidence scores observed for the copies of an image masked with a setof binary masks (M:Λ→{0,1}) are used to form the explanation map by,

S _(I,Ψ)(λ)=

_(M)[Ψ(I⊙m)|m(λ)=1]  Equation 9:

where I⊙m denotes a masked image obtained by pointwise multiplicationbetween the input image and a mask mϵM. The representation of equation 9can be modified to be generalized for sets of smooth masks (M:Λ→[0,1]).Hence, Equation 9 may be reformatted as:

S _(I,Ψ)(λ)=

_(M)[Ψ(I⊙m)·C _(m)(λ)]  Equation 10:

where the term C_(m)(λ) indicates the contribution amount of each pixelin the masked image. Setting the contribution indicator asC_(m)(λ)=m(λ), makes equation 10 equivalent to equation 9. These scoresmay be normalized according to the size of the perturbation masks todecrease the assigned reward to the background pixels when a high scoreis reached for a mask with too many activated pixels. Thus, this termmay be defined as:

$\begin{matrix}{{C_{m}(\lambda)} = {\frac{m(\lambda)}{\sum_{\lambda \in \Lambda}{m(\lambda)}}:}} & {{Equation}\mspace{14mu} 11}\end{matrix}$

Such a formulation may increase the concentration on smaller features,particularly when multiple objects (either from the same instance ordifferent ones) are present in an image.

Putting block-wise layer selection policy and attribution mask selectionstrategy together with an existing framework such as from the RISEmethod, for each CNN containing B convolutional blocks, the last layerof each block is indicated as l_(b)ϵ{1, . . . , B}. Using equations 10and 11, corresponding visualization maps may be formed for each of theselayers by:

$\begin{matrix}{{V_{I,\Psi}^{(l_{b})}(\lambda)} = {{\mathbb{E}}_{M_{d}^{(l_{b})}}\left\lbrack {{\Psi\left( {I \odot m} \right)} \cdot {C_{m}(\lambda)}} \right\rbrack}} & {{Equation}\mspace{14mu} 12}\end{matrix}$

Phase 4: Feature Aggregation

In the fourth phase, the flow of features from low-level to high-levelblocks may be tracked using a fusion module 600, as shown in FIG. 6. Theinputs to the fusion module may be the visualization layers 601-605obtained from Phase Three, above. The fusion module is responsible forcorrecting spatial distortions caused by upsampling coarse feature mapsto higher dimensions and refining the localization of attributionsderived from the model.

Specifically, with respect to FIG. 6, the fusion module includescascaded fusion blocks. First, an unweighted addition is performed fortwo consecutive visualization layers which aggregates and sums up theinformation in the visualization layers—representing two consecutiveconvolutional blocks of the network. Then, a combination of otsu-basedbinarization and pointwise multiplication is performed, whereby featuresthat are present in a current block, but are discarded and/or notincluded in a next block, are eliminated. The features that are absentin the latter visualization map are removed from the collectiveinformation by masking the output of the addition block with a binarymask indicating the activated regions in the latter visualization map.

As an example, referring to block 3, the visualization map 603 mayinclude some shapes or edges which correspond to background imageinformation, such as a sofa or table depicted in the background of theimage of the dog of the input image (see 501 of FIG. 5). However, assuch information of the background image is not included in thevisualization map 604 of block 4, the otsu-based binarization andpointwise multiplication operations act to remove such information fromvisualization map 603. As shown in FIG. 6, the above operations areiterated for all visualization maps of all of the blocks, resulting inan explanation map 606 which represents and emphasizes features that arecommon and present in all of the input visualization maps 601-605. Inthis way, the cascading fusion blocks allow the features determining themodel's prediction to be represented in a more focused manner while theinexplicit features are discarded.

With reference to the fusion module 600, it is noted that in someembodiments a weight may be applied to the visualization maps to focuson the features prevalent in either the earlier or later blocks.However, in the embodiment shown in FIG. 6, no weighting is included inthe input of the visualization maps 601-605 and generation of the finalexplanation map 606.

As discussed, the resulting explanation map 606 is a 2D image having thesame dimensionality as the input image, where regions of the image areindicated with different colors or patterns to indicate to a user theportions of the input image which contribute to the final decisionmaking of the CNN in an image classification task, for example.

Referring now to FIG. 7, a flowchart showing an example of a method 700of the present disclosure is shown. Starting at 701, an input image isinput to a CNN to obtain a classification determination. For example,the input image 501 shown in FIG. 5 may be input to a CNN to obtain aclassification result of “dog” having a certain confidence score of theCNN. At 702, sets of the feature maps that are output from each of thepooling layers of the CNN are extracted. As discussed with respect toPhase 1, most of the information in each model can be collected byprobing the output of the pooling layers, thus visualizing the output ofthe pooling layers allows for tracking the way features are propagatedthrough the convolutional blocks of the CNN.

Based on the extracted sets of feature maps, at 703, a subset of each ofthe pooling layer sets are selected as being the most deterministicfeature maps among each extracted set. As discussed with respect toPhase 2, backward propagation is performed to a particular layer toscore the average gradient of the model's confidence score to each ofthe feature maps, resulting in a value representing the correlationbetween each particular feature map and the original input image. Thesubset is selected based on the gradient of each feature set, where anexample of the selection threshold may selection of only positivegradient feature maps.

At 704, attribution masks are generated by upsampling and processing thesubset of feature maps, as shown for example in FIG. 5B. The attributionmasks are then applied to copies of the input image and the perturbedinput image is input to the CNN, where the confidence score of the CNNwith the perturbed input is used to generate visualization maps for eachlayer of the CNN, shown at 706.

Finally at 707, the visualization maps are input to a fusion module, forexample as discussed with respect to FIG. 6, to combine features of thevisualization maps to generate the final explanation map. The finalexplanation map may represent a heatmap, for example shown at 606 ofFIG. 6, showing the regions of the input image which have the highestdegree of contribution to the decision making of the CNN, which iseasily interpretable by a user.

Testing Results

Based on the embodiments of the present disclosure discussed above,results of experiments showing advantages over the existing art areprovided in the following discussion. The performance of animplementation of the present disclosure has been tested on shallow anddeep CNNs, including VGG16, ResNet-50, and ResNet-101 architectures. Toconduct the experiments, PASCAL VOC 2007 and Severstal datasets wereemployed. The former is a popular object detection dataset containing4,952 test images belonging to 20 object classes. As images with manysmall object occurrences and multiple instances of different classes areprevalent in this dataset, it would be difficult for an explanationartificial intelligence algorithm to perform well on the whole dataset.The latter is an industrial steel defect detection dataset created foranomaly detection and steel defect segmentation problems, which wasreformatted into a defect classification dataset instead, containing11,505 test images from 5 different classes, including one normal classand four different defects classes. Class imbalance, intraclassvariation, and interclass similarity were the main challenges of thisrecast dataset.

Experiments conducted on the PASCAL VOC 2007 dataset were evaluated onits test set with a VGG16, and a ResNet50 model from the TorchRaylibrary, both trained for multi-label image classification. The top-5accuracies of the models on the test set are 93.29% and 93.09%,respectively. On the other hand, for conducting experiments onSeverstal, a ResNet-101 model was trained (with a test accuracy of86.58%) on the recast dataset to assess the performance of the presentdisclosure in the task of visual defect inspection. To recast theSeverstal dataset for classification, the training and test images werecropped into patches of size 256×256. In the evaluations, a balancedsubset of 1,381 test images belonging to defect classes labeled as 1, 2,3, and 4 was chosen. The embodiments of the present disclosure wereimplemented on Keras and the parameter was set to its default value, 0.

With respect to qualitative results of testing, implementation of thepresent disclosure was compared with other existing art methods onsample images from the Pascal dataset, as shown in FIG. 8, and theSeverstal dataset, as shown in FIG. 9. Referring to FIGS. 8 and 9, thevarious explanation maps generated by known existing methods arecompared to the explanation map generated by implementation of thepresent disclosure 802, 902. Images with both normal-sized and smallobject instances are included in the testing along with theircorresponding confidence scores. For example, in FIG. 8, an input imageof a cat shown in a large size within the image, or an input image of atrain on train tracks from a long distance perspective, resulting in asmall image of the train within the input image. As shown in FIG. 8, theexplanation maps resulting from implementation of the present disclosureincludes a heatmap wherein the “hot”, or high impact areas of theheatmap, are smaller and more focused on the areas of the input imagewhich are used by the CNN for the classification output. Even fordefective classifications, as shown in FIG. 9, the resulting explanationmap provides more information to a user as to the network'sclassification determination. This provides more useful information tothe user in evaluating and understanding the decision making of thenetwork.

This is further shown in FIG. 10, which shows the class discriminativeability of the present disclosure given an input image havingmulti-class classification results. FIG. 10 shows an input image 1000which includes an image of both a person and a motorbike, resulting intwo classification results of “person” and “motorbike” with confidencescores of 0.9928 and 0.0071, respectively. Given the input image 1000,existing methods of the prior art, such as RISE, generate explanationmaps 1001, 1002 which lack class discriminative ability. The RISEexplanation maps 1001, 1002 include a heat map that is not focused onparticular or specific features or regions of the input image, and thusthe explanation map does not provide much useful information to a userin gleaning insight on the network's classification decision.

By contrast, the explanation maps 1003, 1004 of the present disclosureare clearly indicative of the exact regions of the input image whichcontribute the most to the network's classification decision where, forthe classification “motorbike”, the hot areas of the heatmap clearlycorrespond to the features of the motorcycle, and where, for theclassification “person”, the hot areas of the heatmap clearly correspondto the features of the person's head, face, hand, and arm areas. Thus,the superior ability of embodiments of the present disclosure indiscriminating the explanations of various classes in comparison withexisting methods, such RISE for example, can be clearly appreciated.

With respect to quantitative results of testing, evaluation of resultsis categorized into “ground truth-based” and “model truth-based”metrics. The former is used to justify the model by assessing the extentto which the algorithm satisfies the users by providing visuallysuperior explanations, while the latter is used to analyze the model'sbehavior by assessing the faithfulness of the algorithm and itscorrectness in capturing the attributions in line with the model'sprediction procedure. The results of existing methods of the prior artin comparison to those of the present disclosure (SISE) are shown inFIG. 11. The utilized metrics are discussed below.

Ground truth-based Metrics: The explanation algorithms of the existingart are compared with SISE based on three distinct ground-truth basedmetrics to justify the visual quality of the explanation maps generatedby the present disclosure. Denoting the ground-truth mask as G and theachieved explanation map as S, the evaluation metrics used are asfollows:

“Energy-Based Pointing Game (EBPG)” evaluates the precision anddenoising ability of Explainable AI algorithms. Extending thetraditional Pointing Game, EBPG considers all pixels in the resultantexplanation map S for evaluation by measuring the fraction of its energycaptured in the corresponding ground truth G, as

${EBPG} = {\frac{{{S \odot G}}_{1}}{{S}_{1}}.}$

“mIoU” analyses the localization ability and meaningfulness of theattributions captured in an explanation map. In the testing, the top 20%pixels highlighted in each explanation map S are selected and the meanintersection over union is computed with their correspondingground-truth masks.

“Bounding box (Bbox)” is taken into account as a size-adaptive variantof mIoU. Considering N as the number of ground truth pixels in G, theBbox score is calculated by selecting the top N pixels in S andevaluating the corresponding fraction captured over G.

Model truth-based metrics: To evaluate the correlation between therepresentations of the present disclosure and the model's predictions,model-truth based metrics are employed to compare implementations of thepresent disclosure with the methods of the existing art. As visualexplanation algorithms' main objective is to envision the model'sperspective for its predictions, these metrics are considered of higherimportance.

“Drop %” and “Increase %”, may be interpreted as an indicator of thepositive attributions missed and the negative attribution discarded fromthe explanation map, respectively. Given a model Ψ(.), an input imageI_(i) from a dataset containing K images, and an explanation mapS(I_(i)), the Drop/Increase % metric selects the most important pixelsin S(I₃) to measure their contribution towards the model's prediction. Athreshold function T(.) is applied on S(I_(i)) to select the top 15%pixels that are then extracted from I_(i) using pointwise multiplicationand fed to the model. The confidence scores on such perturbed images arethen compared with the original score, according to the equations:

$\begin{matrix}{{{{Drop}\mspace{14mu}\%} = {\frac{1}{K}{\sum_{i = 1}^{K}{\frac{\max\left( {0,{{\Psi\left( I_{i} \right)} - {\Psi\left( {I_{i} \odot {T\left( I_{i} \right)}} \right)}}} \right)}{\Psi\left( I_{i} \right)} \times 100}}}}{{{Increase}\mspace{14mu}\%} = {\sum_{i = 1}^{K}{{sign}\left( {{\Psi\left( {I_{i} \odot {T\left( I_{i} \right)}} \right)} - {\Psi\left( I_{i} \right)}} \right)}}}} & \;\end{matrix}$

The experimental results discussed above with respect to FIGS. 8, 9, 10demonstrate the resolution, and concreteness of explanation mapsgenerated according to the present disclosure, which is furthersupported by ground truth-based evaluation metrics as in FIG. 11. Also,model truth-based metrics in FIGS. 11 and 12 prove the presentdisclosure's supremacy in highlighting the evidence, based on which themodel makes a prediction. (In FIGS. 11 and 12, a lower Drop % is better,and a higher Increase % is better). Similar to the CAM-based methods,the output of the last convolutional block plays the most critical rolein the present disclosure. However, by considering the intermediatelayers based on the block-wise layer selection, SISE's advantageousproperties are enhanced. Furthermore, utilizing attribution-based inputsampling instead of a randomized sampling, ignoring the unrelatedfeature maps, and modifying the linear combination step dramaticallyimproves the visual clarity and completeness offered by the embodimentsaccording to the present disclosure.

In addition to the above performance evaluations, a runtime test is alsoprovided to compare the complexity of the present disclosure with theexisting methods, specifically using a Tesla T4 GPU with 16 GB of memoryand the ResNet-50 model. Reported runtimes were averaged over 100 trialsusing random images from the PASCAL VOC 2007 test set. Other thanGRAD-CAM and GRAD-CAM++ which achieved the best runtimes—only due to thefact that these algorithms only require a single forward pass and asingle backward pass—Extremal Perturbation recorded the longest runtime,78.37 seconds, since it optimizes numerous variables. In comparison withRISE, which has a runtime of 26.08 seconds, SISE of the presentdisclosure runs in 9.21 seconds.

Further, while RISE of the existing art uses around 8000 random masks tooperate on a ResNet-50 model, SISE of the present disclosure uses around1900 attribution masks with μ set to 0, out of a total of 3904 featuremaps initially extracted from the same ResNet-50 model beforenegative-gradient feature maps were removed. The difference in thenumber of masks allows SISE to operate in around 9.21 seconds. Toanalyze the effect of reducing the number of attribution masks on SISE'sperformance, an ablation study is provided. By changing μ to 0.3, ascanty variation in the boundary of explanation maps can be noticedwhile the runtime is reduced to 2.18 seconds. This shows that ignoringfeature maps with low gradient values does not considerably affect SISEoutputs since they tend to be assigned low scores in the third phase ofSISE anyway. By increasing μ to 0.5, a slight decline in the performancewas recorded along with a runtime of just 0.65 seconds.

Referring now to FIG. 13, an illustration of an example computer 1300 isprovided which may be used to embody, implement, execute, or performembodiments of the present disclosure. In selected embodiments, thecomputer 1300 may include a bus 1303 (or multiple buses) or othercommunication mechanism, a processor 1301, processor internal memory1301 a, main memory 1304, read only memory (ROM) 1305, one or moreadditional storage devices 1306, and/or a communication interface 1302,or the like or sub-combinations thereof. The embodiments describedherein may be implemented within one or more application specificintegrated circuits (ASICs), digital signal processors (DSPs), digitalsignal processing devices (DSPDs), programmable logic devices (PLDs),field programmable gate arrays (FPGAs), processors, controllers,micro-controllers, microprocessors, other electronic units designed toperform the functions described herein, or a selective combinationthereof. In all embodiments, the various components described herein maybe implemented as a single component, or alternatively may beimplemented in various separate components.

A bus 1303 or other communication mechanism, including multiple suchbuses or mechanisms, may support communication of information within thecomputer 1300. The processor 1301 may be connected to the bus 1303 andprocess information. In selected embodiments, the processor 1301 may bea specialized or dedicated microprocessor configured to performparticular tasks in accordance with the features and aspects disclosedherein by executing machine-readable software code defining theparticular tasks. In some embodiments, multiple processors 1301 may beprovided with each processing unit dedicated to a particular specializedtask, such as graphics processing or artificial intelligence relatedprocessing.

Main memory 1304 (e.g., random access memory—or RAM—or other dynamicstorage device) may be connected to the bus 1303 and store informationand instructions to be executed by the processor 1301. Processor 1301may also include internal memory 1301 a, such as CPU cache implementedby SRAM, for storing data used for executing instructions. Utilizationof internal memory 1301 a may optimize data and memory management byreducing memory bandwidth usage with main memory 1304. Although FIG. 13depicts internal memory 1301 a as a component of processor 1301, it willbe understood that embodiments are included wherein internal memory 1301a is a separate component apart from processor 1301. Main memory 1304may also store temporary variables or other intermediate informationduring execution of such instructions.

ROM 1305 or some other static storage device may be connected to a bus1303 and store static information and instructions for the processor1301. An additional storage device 1306 (e.g., a magnetic disk, opticaldisk, memory card, or the like) may be connected to the bus 1303. Themain memory 1304, ROM 1305, and the additional storage device 1306 mayinclude a non-transitory computer-readable medium holding information,instructions, or some combination thereof, for example instructions thatwhen executed by the processor 1301, cause the computer 1300 to performone or more operations of a method as described herein. A communicationinterface 1302 may also be connected to the bus 1303. A communicationinterface 1302 may provide or support two-way data communication betweena computer 1300 and one or more external devices (e.g., other devicescontained within the computing environment).

In selected embodiments, the computer 1300 may be connected (e.g., via abus) to a display 1307. The display 1307 may use any suitable mechanismto communicate information to a user of a computer 1300. For example,the display 1307 may include or utilize a liquid crystal display (LCD),light emitting diode (LED) display, projector, or other display deviceto present information to a user of the computer 1300 in a visualdisplay. One or more input devices 1308 (e.g., an alphanumeric keyboard,mouse, microphone, stylus pen) may be connected to the bus 1303 tocommunicate information and commands to the computer 1300. In selectedembodiments, one input device 1308 may provide or support control overthe positioning of a cursor to allow for selection and execution ofvarious objects, files, programs, and the like provided by the computer1300 and displayed by the display 1307.

The computer 1300 may be used to transmit, receive, decode, display, orthe like one or more image or video files. In selected embodiments, suchtransmitting, receiving, decoding, and displaying may be in response tothe processor 1301 executing one or more sequences of one or moreinstructions contained in main memory 1304. Such instructions may beread into main memory 1304 from another non-transitory computer-readablemedium (e.g., a storage device).

Execution of sequences of instructions contained in main memory 1304 maycause the processor 1301 to perform one or more of the procedures orsteps described herein. In selected embodiments, one or more processorsin a multi-processing arrangement may also be employed to executesequences of instructions contained in main memory 1304. Alternatively,or in addition thereto, firmware may be used in place of, or inconnection with, software instructions to implement procedures or stepsin accordance with the features and aspects disclosed herein. Thus,embodiments in accordance with the features and aspects disclosed hereinmay not be limited to any specific combination of hardware circuitry andsoftware.

Non-transitory computer readable medium may refer to any medium thatparticipates in holding instructions for execution by the processor1301, or that stores data for processing by a computer, and comprise allcomputer-readable media, with the sole exception being a transitory,propagating signal. Such a non-transitory computer readable medium mayinclude, but is not limited to, non-volatile media, volatile media, andtemporary storage media (e.g., cache memory). Non-volatile media mayinclude optical or magnetic disks, such as an additional storage device.Volatile media may include dynamic memory, such as main memory. Commonforms of non-transitory computer-readable media may include, forexample, a hard disk, a floppy disk, magnetic tape, or any othermagnetic medium, a CD-ROM, DVD, Blu-ray or other optical medium, RAM,PROM, EPROM, FLASH-EPROM, any other memory card, chip, or cartridge, orany other memory medium from which a computer can read.

In selected embodiments, a communication interface 1302 may provide orsupport external, two-way data communication to or via a network link.For example, a communication interface 1302 may be a wireless networkinterface controller or a cellular radio providing a data communicationnetwork connection. Alternatively, a communication interface 1302 maycomprise a local area network (LAN) card providing a data communicationconnection to a compatible LAN. In any such embodiment, a communicationinterface 1302 may send and receive electrical, electromagnetic, oroptical signals conveying information.

A network link may provide data communication through one or morenetworks to other data devices (e.g., other computers such as 1300, orterminals of various other types). For example, a network link mayprovide a connection through a local network of a host computer or todata equipment operated by an Internet Service Provider (ISP). An ISPmay, in turn, provide data communication services through the Internet.Accordingly, a computer 1300 may send and receive commands, data, orcombinations thereof, including program code, through one or morenetworks, a network link, and communication interface 1302. Thus, thecomputer 1300 may interface or otherwise communicate with a remoteserver, or some combination thereof.

The various devices, modules, terminals, and the like discussed hereinmay be implemented on a computer by execution of software comprisingmachine instructions read from computer-readable medium, as discussedabove. In certain embodiments, several hardware aspects may beimplemented using a single computer, in other embodiments multiplecomputers, input/output systems and hardware may be used to implementthe system.

For a software implementation, certain embodiments described herein maybe implemented with separate software modules, such as procedures andfunctions, each of which perform one or more of the functions andoperations described herein. The software codes can be implemented witha software application written in any suitable programming language andmay be stored in memory and executed by a controller or processor.

The foregoing disclosed embodiments and features are merely exemplaryand are not to be construed as limiting the present invention. Thepresent teachings can be readily applied to other types of apparatusesand processes. The description of such embodiments is intended to beillustrative, and not to limit the scope of the claims. Manyalternatives, modifications, and variations will be apparent to thoseskilled in the art.

What is claimed is:
 1. A method for outputting an explanation map for anoutput determination of a convolutional neural network (CNN) based on aninput image, the method comprising: extracting a plurality of sets offeature maps from a corresponding plurality of pooling layers of theCNN; obtaining a plurality of attribution masks based on subsets of theplurality of sets of feature maps; applying the plurality of attributionmasks to copies of the input image to obtain a plurality of perturbedinput images; obtaining a plurality of visualization maps based onconfidence scores by inputting the plurality of perturbed copies of theinput image to the CNN; and outputting an explanation map of the outputdetermination of the CNN based on the plurality of visualization maps.2. The method of claim 1, further comprising identifying the mostdeterministic feature maps with regard to the input image of each of theplurality of sets of feature maps as a corresponding subset of featuremaps.
 3. The method of claim 2, wherein identifying the mostdeterministic feature maps comprises calculating an average gradient ofthe model's confidence score with respect to the input image for eachfeature map, and wherein a feature map is identified as mostdeterministic based on the feature map having a corresponding averagegradient that is greater than zero.
 4. The method of claim 2, furthercomprising upscaling the feature maps of each subset to an original sizeof the input image to generate the plurality of attribution masks. 5.The method of claim 4, wherein generating the plurality of attributionmasks further comprises performing linear transformation on theattribution mask that normalizes the values in the range of [0, 1]. 6.The method of claim 1, wherein applying the generated attribution masksto the copies of the input image comprises performing perturbation ofthe copies of the input image based on each of the generated attributionmasks by point-wise multiplication.
 7. The method of claim 1, whereinoutputting the explanation map comprises performing a fusion process tocombine feature information from the plurality of visualization maps. 8.The method of claim 7, wherein the fusion process to combine featureinformation of visualization maps of the plurality of visualization mapscomprises: normalizing a first visualization map of the plurality ofvisualization maps; performing unweighted addition of the normalizedfirst visualization map and a normalized second visualization map toobtain a first result; performing Otsu-based binarization on thenormalized second visualization map to eliminate features which are notpresent in the normalized first visualization map to obtain a secondresult; performing point-wise multiplication on the first result and thesecond result to obtain a third result; and performing the fusionprocess using the third result and a next visualization map of theplurality of visualization maps.
 9. The method of claim 1, wherein theexplanation map has a same dimensionality as the input image.
 10. Amachine-readable non-transitory medium having stored thereonmachine-executable instructions for outputting an explanation map for anoutput determination of a convolutional neural network (CNN) based on aninput image, the instructions comprising: extracting a plurality of setsfeature maps from a corresponding plurality of pooling layers of theCNN; obtaining a plurality of attribution masks based on subsets of theplurality of sets of feature maps; applying the plurality of attributionmasks to copies of the input image to obtain a plurality of perturbedinput images; obtaining a plurality of visualization maps based onconfidence scores by inputting the plurality of perturbed copies of theinput image to the CNN; and outputting an explanation map of the outputdetermination of the CNN based on the plurality of visualization maps.11. The machine-readable non-transitory medium of claim 10, wherein theinstructions further comprise identifying the most deterministic featuremaps with regard to the input image of each of the plurality of sets offeature maps as a corresponding subset of feature maps.
 12. Themachine-readable non-transitory medium of claim 11, wherein identifyingthe most deterministic feature maps comprises calculating an averagegradient of the model's confidence score with respect to the input imagefor each feature map, and wherein a feature map is identified as mostdeterministic based on the feature map having a corresponding averagegradient that is greater than zero.
 13. The machine-readablenon-transitory medium of claim 11, wherein the instructions furthercomprise upscaling the feature maps of each subset to an original sizeof the input image to generate the plurality of attribution masks. 14.The machine-readable non-transitory medium of claim 13, whereingenerating the plurality of attribution masks further comprisesperforming linear transformation on the attribution mask that normalizesthe values in the range of [0, 1].
 15. The machine-readablenon-transitory medium of claim 10, wherein applying the generatedattribution masks to the copies of the input image comprises performingperturbation of the copies of the input image based on each of thegenerated attribution masks by point-wise multiplication.
 16. Themachine-readable non-transitory medium of claim 10, wherein outputtingthe explanation map comprises performing a fusion process to combinefeature information from the plurality of visualization maps.
 17. Themachine-readable non-transitory medium of claim 16, wherein the fusionprocess to combine feature information of visualization maps of theplurality of visualization maps comprises: normalizing a firstvisualization map of the plurality of visualization maps; performingunweighted addition of the normalized first visualization map and anormalized second visualization map to obtain a first result; performingOtsu-based binarization on the normalized second visualization map toeliminate features which are not present in the normalized firstvisualization map to obtain a second result; performing point-wisemultiplication on the first result and the second result to obtain athird result; and performing the fusion process using the third resultand a next visualization map of the plurality of visualization maps. 18.The machine-readable non-transitory medium of claim 10, wherein theexplanation map has a same dimensionality as the input image.
 19. Asystem for outputting an explanation map for an output determination ofa convolutional neural network (CNN) based on an input image, the systemcomprising: a display; one or more processors; and a memory havinginstructions stored thereon, which when executed by the one or moreprocessors, cause the one or more processors to: extract a plurality ofsets of feature maps from a corresponding plurality of pooling layers ofthe CNN; obtain a plurality of attribution masks based on subsets of theplurality of sets of feature maps; apply the plurality of attributionmasks to copies of the input image to obtain a plurality of perturbedinput images; obtain a plurality of visualization maps based onconfidence scores by inputting the plurality of perturbed copies of theinput image to the CNN; and output, via the display, an explanation mapof the output determination of the CNN based on the plurality ofvisualization maps.